Codecov Report

❌ Patch coverage is 4.94580% with 2806 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.84%. Comparing base (74af9b9) to head (4b1fcec).
⚠️ Report is 2 commits behind head on main.

@@             Coverage Diff              @@
##               main   #12758      +/-   ##
============================================
- Coverage     17.92%   17.84%   -0.09%     
- Complexity    16176    16228      +52     
============================================
  Files          5949     6001      +52     
  Lines        534058   537981    +3923     
  Branches      65301    65650     +349     
============================================
+ Hits          95742    95994     +252     
- Misses       427560   431204    +3644     
- Partials      10756    10783      +27     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@JoaoJandre just heads up - my colleagues have been working on an incremental backup feature for NAS B&R (using nbd/qemu bitmap tracking & checkpoints). We're also working on a new Veeam-KVM integration for CloudStack whose PR may may be out soon. My colleagues can further help review and advise on this.

/cc @weizhouapache @abh1sar @shwstppr @sureshanaparti @DaanHoogland @harikrishna-patnala

Just my 2cents on the design & your comments - NAS is more than just NFS, but any (mountable) shared storage such as CephFS, cifs/samba etc. Enterprise users usually don't want to mix using secondary storage with backup repositories, which is why NAS B&R introduced a backup-provider agnostic concept of backup repositories which can be explored by other backup providers.

Just my 2cents on the design & your comments - NAS is more than just NFS, but any (mountable) shared storage such as CephFS, cifs/samba etc.

At the time of writing that part, I believe it was only NFS that was supported. I'll update the relevant part.

Enterprise users usually don't want to mix using secondary storage with backup repositories, which is why NAS B&R introduced a backup-provider agnostic concept of backup repositories which can be explored by other backup providers.

The secondary storage selector feature (introduced in 2023 by #7659) allows you to specialize secondary storages. This PR extended the feature so that you may also create selectors for backups.

Hi Joao,

This looks promising. Incremental backups, quick restore and file restore features have been missing from CloudStack KVM.

I am having trouble understanding some of the design choices though:

1. What’s the reason behind strong coupling with secondary storage?

   • I am wondering if the Backup Repository will provide a more flexible alternative. The user would be free to add an external storage server or use the secondary storage by simply adding it as a backup repository? It will be very easy for user to have multiple backup repository attached to multiple backup offerings which can be assigned to instance as required.

     This will also be consistent with other backup providers like Veeam and NAS which have the concept of backup repository.

     The backup repository feature also comes with a separate capacity tracking and email alerts.

   • If a secondary storage is needed just for backup’s purpose, how will it be ensured that templates and snapshots are not copied over to it?

2. About Qemu compression

   • Have you measured / compared the performance of qemu-img compression with other compression methods?
   • As I understand, qemu-img compresses the qcow2 file at a cluster granularity (usually 64kb). That might not fare well when compared to storage level compression. In production environments, the operator might choose to have compression at the storage layer if they are using an enterprise storage like NetApp. Even something open source like ZFS might perform better than qemu-img compress due to the granularity limitation that qemu compression has.
   • I am making this point because the compression part is introducing a fair bit of complexity due to the interaction with SSVM, and I am just wondering if the gains are worth the trouble and should compression be offloaded to the storage completely.

3. Do we need a separate backup offering table and api?

   • Why not add column or details to backup_offering or backup_offering_details? Other offerings can also benefit from these settings.

4. What’s the reason behind using virDomainSnapshotCreate to create backup files and not virDomainBackupBegin like incremental volume snapshots and NAS backup?

   • Did you face any issues with checkpoints and bitmaps?

Hi Joao,

Hello, @abh1sar

This looks promising. Incremental backups, quick restore and file restore features have been missing from CloudStack KVM.

I am having trouble understanding some of the design choices though:

1. What’s the reason behind strong coupling with secondary storage?
   
   * I am wondering if the Backup Repository will provide a more flexible alternative. The user would be free to add an external storage server or use the secondary storage by simply adding it as a backup repository? It will be very easy for user to have multiple backup repository attached to multiple backup offerings which can be assigned to instance as required.

I don't see why we should force the coupling of backup offerings with backup repositories, what is the benefit?

     This will also be consistent with other backup providers like Veeam and NAS which have the concept of backup repository.
     The backup repository feature also comes with a separate capacity tracking and email alerts.

The secondary storage also has both features. Although the capacity is not reported to the users currently.

   * If a secondary storage is needed just for backup’s purpose, how will it be ensured that templates and snapshots are not copied over to it?

The secondary storage selectors feature (introduced in 2023 through #7659) allows you to specialize secondary storages. Quoting from the PR description: "This PR aims to add the possibility to direct resources (Volumes, Templates, Snapshots and ISOs) to a specific secondary storage through rules written in JavaScript that will only affect new allocated resources". For a few years it has been possible to have secondary storages that only receive snapshots or templates for example. This PR introduces the possibility to add selectors for backups, so that you have secondary storages that are specific for backups.

Furthermore, my colleagues are working on a feature to allow using alternative secondary storage solutions, such as CephFS, iSCSI and S3, while preserving compatibility with features destined to NFS storages. This feature may be extended in the future to allow essentially any type of secondary storage. Thus, the flexibility for secondary storages will soon grow.

2. About Qemu compression
   
   * Have you measured / compared the performance of qemu-img compression with other compression methods?

Using any other type of backup-level compression will be worse then using qemu-img compression. This is because when restoring the backup, we must have access to the whole backing chain. If we use other types of compression, we will have to decompress the whole chain before restoring. Using qemu-img, the backing files are still valid and do not need to be decompressed, we actually never have to decompress ever. This is the great benefit of using qemu-img.

In any case, here is a brief comparison of using qemu-img with the zstd library and 8 threads and using the pigz implementation of multi-threaded compression, also using 8 threads. The original file is the root volume of a VM that I use.

Compression using qemu-img was a lot faster, with a bit smaller compression ratio. Furthermore, we have to consider that the qemu-img compressed image can be used as-is, while the other images must be decompressed, further adding to the processing time of backing up/restoring a backup.

   * As I understand, qemu-img compresses the qcow2 file at a cluster granularity (usually 64kb). That might not fare well when compared to storage level compression. In production environments, the operator might choose to have compression at the storage layer if they are using an enterprise storage like NetApp. Even something open source like ZFS might perform better than qemu-img compress due to the granularity limitation that qemu compression has.

The compression feature is optional, if you are using storage-level compression, you probably will not use backup-level compression. However, many environments do not have storage-level compression, thus having the possibility of backup-level compression is still very interesting.

   * I am making this point because the compression part is introducing a fair bit of complexity due to the interaction with SSVM, and I am just wondering if the gains are worth the trouble and should compression be offloaded to the storage completely.

The compression does not add any interaction with the SSVM.

3. Do we need a separate backup offering table and api?
   
   * Why not add column or details to backup_offering or backup_offering_details? Other offerings can also benefit from these settings.

I did not want to add dozens of parameters to the import backup offering API which are only really going to be used for one provider. This way, the original design of the API is preserved.

Furthermore, you may note that the APIs are intentionally not called createKnibBackupOffering, but createNativeBackupOffering. If other native providers want to use these offerings, they may do so by extending their implementations.

4. What’s the reason behind using virDomainSnapshotCreate to create backup files and not virDomainBackupBegin like incremental volume snapshots and NAS backup?
   
   * Did you face any issues with checkpoints and bitmaps?

There are two main issues with using bitmaps:

1. They are prone to corruption, while this can be mitigated in some ways, since the incremental volume snapshot feature was added, we have noticed multiple cases of bitmap corruption with different causes. It is possible to detect the corruption and delete corrupt bitmaps, but this would add more complexity to the feature.
2. Using bitmaps is not compatible with the file-based incremental VM snapshot feature added in File-based disk-only VM snapshot with KVM as hypervisor #10632. After some internal discussion and feedback from users, we have come to the conclusion that being able to use both the incremental VM snapshot and backup features at the same time is very interesting.

At the end of the day, this PR adds a new backup provider option for users. They will be free to choose the provider that best fits their needs. This is one of the reasons why it was done as a new backup provider; KNIB and other backup providers do not have to cancel each-other out.

Hi @JoaoJandre
Please find my response inline.

I don't see why we should force the coupling of backup offerings with backup repositories, what is the benefit?

It has a big benefit for use cases where someone wants multiple tiers of backup storage with different cost and recovery characteristics. For example, long-term archival backups might go to cheap cold storage like tape, while backups that require faster recovery (better RTO) may use more expensive SSD-backed storage.

You can have multiple backup offerings for different use cases and VMs can be attached to the required offerings.
Furthermore, the backup storage usage is tracked per Backup Offering. So, users can use different offerings as per requirement and get charged accordingly.

The secondary storage also has both features. Although the capacity is not reported to the users currently.

Capacity is just for Admins for Backup Repository also, so that is not the issue.
But the Secondary Storage capacity tracking and alerts would be mixed up for Templates, Snapshots, Volumes and Backups.
With backup repositories, you will get separate capacity tracking and alerts just for the Backup Infrastructure.

Compression using qemu-img was a lot faster, with a bit smaller compression ratio. Furthermore, we have to consider
that the qemu-img compressed image can be used as-is, while the other images must be decompressed, further adding
to the processing time of backing up/restoring a backup.

It does have its benefits, but do keep in mind that the decompression cost will be paid by Reads. Also if someone is using a restored VM, they might see the physical size increase disproportionately to the actual data written due to decompression. It's still a useful feature and it's good that it is optional.

I did not want to add dozens of parameters to the import backup offering API which are only really going to be used for one provider. This way, the original design of the API is preserved.

If there is possibility these parameters can be added for other providers they should be added to the existing API.
Some of these like allowQuickRestore, allowExtractFile might as well be added for other providers such as Veeam and NAS in the future.

Furthermore, you may note that the APIs are intentionally not called createKnibBackupOffering, but createNativeBackupOffering. If other native providers want to use these offerings, they may do so by extending their implementations.

• What defines a provider as Native?
• Do the native offerings differ in functionality or usage to the existing BackupOfferings?
• How will this look like in the UI? Will the user see Backup Offering and another Native Backup Offering?
• Again, coupling an offering to storage has its benefits which current Backup Offerings already have.

Hello, @abh1sar

It has a big benefit for use cases where someone wants multiple tiers of backup storage with different cost and recovery characteristics. For example, long-term archival backups might go to cheap cold storage like tape, while backups that require faster recovery (better RTO) may use more expensive SSD-backed storage.

You can have multiple backup offerings for different use cases and VMs can be attached to the required offerings. Furthermore, the backup storage usage is tracked per Backup Offering. So, users can use different offerings as per requirement and get charged accordingly.

You can have exactly that using this implementation. Essentially backup repositories and secondary storages are the same, with different names. Using selectors, you can have offerings that are tied to specific secondary storages, or you can have offerings that go into multiple storages, it was made to be flexible.

Capacity is just for Admins for Backup Repository also, so that is not the issue. But the Secondary Storage capacity tracking and alerts would be mixed up for Templates, Snapshots, Volumes and Backups. With backup repositories, you will get separate capacity tracking and alerts just for the Backup Infrastructure.

Again, you can have dedicated secondary storages using selectors.

If there is possibility these parameters can be added for other providers they should be added to the existing API. Some of these like allowQuickRestore, allowExtractFile might as well be added for other providers such as Veeam and NAS in the future.

While I was not the one that introduced the backup framework, looking at its design, it was clearly intended to have as little configuration as possible on the ACS side while leaving these details to the backup provider. If we add these parameters to the import backup offering API, I'm sure a lot of users will be confused when they do nothing for Veeam and Dell Networker. I did not intend to warp the original design of configuring the offerings on the provider side and only importing it to ACS.

This is why I created the concept of native offerings. As with KNIB (and NAS) the provider is ACS, and thus the configurations can still be made on the "backup provider", and the import API will follow the same logic it always had.

* What defines a provider as `Native`?

A provider is native if the backup implementation is made by ACS. Thus, the current native providers would be NAS and KNIB. Veeam and Networker are external providers.

* Do the native offerings differ in functionality or usage to the existing BackupOfferings?

They are used to configure the details that would be configured in the backup provider if it was an external provider, such as Veeam for example.

* How will this look like in the UI? Will the user see `Backup Offering` and another `Native Backup Offering`?

I have yet to add it to the GUI, if you have any suggestions for the GUI design, you are free to give your opinion. The GUI for the native backup offerings will be added in the future.

* Again, coupling an offering to storage has its benefits which current Backup Offerings already have.

Again, you absolutely can do this with KNIB. But you may also choose not to do it, its flexible.